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A. PURPqSE Except in instruction/:^ . situations, where student 
programming is the g^fim^ry activity, 
microcomputers can *e used to assist the 
ojistxuctional pro9ess effectively only to .the - 



extent that quality software is available, 

This Evaluator ' s ' Guider .>has .been developed to' ' 
provide backgrgTund information and forms to aide 
,teacher§^ ahd other educators' in evaluating* ' 
educational software and courseware. Two forms, 
^Courseware Description" and "Courseware ' . 
Evaluation", are described. '^The design o£ the 
-evaluation form is based on the assumption that 
most of the information on the description, form is 
available. i * ^ , - ' 

The forms were^ based.originally on the 'forms 
developed and used.^y. the CONDUIT Project for ^ 
evaluating compqte^based instruction packages for 
past-seoondary institutions/ with Additional 
concepts adopted from fojcms* developed by other *, ' 
^organization's and individuals, , ^ ' \^ 

The^' Guide wa^^iginally designed to be used by ' 
school* j>ei:sonnel who are 'partieipatifi€|— in the — 
goursewar^ evgiJLuation process of MicroSIFT/ 
'de.fiJcribed**irt the jrffexfe section. In addition. 



however rt^h as J l K J iu I uiind ^a?efrir-lyr^ifKlividualv 
teachers br jpthers who wish to evaluate courseware , ]^ 
before ^purchaaTing, and as a supplement to " ' •% ' 
preservice ^nd inservice courses: concerned, with 
the development or use of computeip based ^ , - . 

applications . : * ' 
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B. PROCESS In December 1979, under ^ contract with the' 
. ' Natidnal Institute of Education, the Computer 
y TPechnology Program of^ the -Northwest Regional 
- ^- -^ijeat-ibnal- ijaboratory beg a n d es igni Tig-a~ — 



clearinghouse for micrpcomputer-based educational* 
software and courseware'. The clearinghouse, 
called MicroSIFT (Microcomputer Software and 
Infc^rmation For Teachers) , .ifas ^s'one of its goals 
the development and implenl?^ntation/of an 
evaluation process and related 'instruments for^ 
such courseware. \ 

The. design (^f MicroSIFT includes a selected 
network of ^significant centers of instructional 
computing activiti^ at^^the K-12 l^vel. The 
cfenters are l^rge school distriyts or regional 
consortia where fujl time instructional computing 
staff exist, and which have a history of 
development, -evaluation* and implementation of ' 
instructional applications of computers^, This* 
Network f6rms the basis for the evaluation process. 

• » « 

The four stages of the evaluation ^process are 
described -on -the following pages. . ^ 
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I. ; INJ^RpQUCTION 
(Continued)', 



B.- PROCESS . " 
(Cpntiriued) 



Phase -I-r- 
Sif ting « 



first look at a ^^ackage which screens, 
out those pj'rograms that are. not 
instructioi^'al in nature. In addition to 
a program's inst^ructional- valu^,\ the 
sifting ph^se determines a package* s^ 
."operational re^c^iness" and its 
, "hardware compatibility" status. The 
'MictoBIFT* staff are ,res^nsible for this 
stage. ■ ' ' . ^ 



2. 'Phase II|. 



Assuming the software fails to meet the 
MlcroSIFT standards for the .sifting ° 
phase, the evaluation terminate. 
' However, if the software qualifies ^as 
."?,nstruetional", "operational? and 

• "compatible"- it, then, enters Phase' II, 
. "Courseware ^Description. " ^ Phase il- 

briefly describes the package specifying 
the prog i;am format, instructional, 
^ujrppse and technique (s) , type of 
package, available documentation ahd the 

• hardware conf iguratfon net^ssary for ~ 
. operation. MicroSlFT staff {complete 

njost of this stage, ustng the . ^ • * 
"Cpurseware Description" form described- 
^in Part li of this. Guide J - 
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(ConCinued) 



B . -PROCESS 

(Continued).' 



Ph.ase III, 
Evaluation 



Te&ehers with experience in the subjeqt 
and grade level of the' material are 
selected from schools served by the 
network site to evaluate qours'ewal 



4. 



Phase IV, 

In-Depth 

EJvaluatioA 



according to. the content, instruct iona 
quality and technic^al quality crite^a 
-^identified in the "Courseware^ 
' ; \Eyal^jation" forW^aescribed in Part III 
of* this Guide. An^*evaluation Is al«o 
, done by, an e^jpert in instructional 
, ^ computing employed at the network 

centex. These reviews are summarized 
- • and published quarterly by MicroSJFT. 

^^-v—^onje .mater i^l^ because o^cpmplexity or 
amouftt of curriculum 'covered, 
warrant more extensive evaluation. : 
Activities in this category would be ^ 
pre- and post-testing pr observation bf 
. student activity at the computer. ' . 
/ instruments to ^uppoct this phase are 
not included in this Guide. 
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I. ' INTRODUCTION 
(Continued) 



c. ' def:^ition of terms 



In the fields of microcomputers and education^ terminology 
can easily be ini^;riterpretedT~So thafe everyone using thife 
Guide ^wi 11 interpret words in the sairie . way, a brief glossary 
is provided be\ow* Each def initioli conforms to some 
standard use of the terin,'but'is not necessarily ^ fcomposite.-^ 
of ail usee nor the mos€ common us^. » , 



Content ; '.Facts, terms, , ideas, concepts, pr^ciples, 
^ ^theories and constructs which m^e up the 
* ^ X subject matter of an instructional package, 

CourseKare. Software and printed m^tderials which support 
' instruction in a CMijaete course of study or 
' * ' a -definable subset^f a* course • In 'this 

- ^ ^6nse, it is npt required that air ' 

\ ' instructional: activities be supported by the 

\ package ^o^hat the ^tutorial mode of i 

* > instru^^xon be employed. Evjeh a 30* minute 

student activity aimed at one objective can ' 
Be^ourseware in this definition. 

.Di3C Operating ^^stem: ^n^operatin^ system 
which includes' the caplbility 9f ,,controlliflg' 
a^l^i <:oordinating* the functions of a disc 
drive within* a Bomputer system. - ' ■ \ 

Graphics Images displayed on a video screen' or printer 

which are generated by a qora'put^r program. 



DOS 



MI 
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C* DEFINITION OF^ TERNiS 
(Contiinu^d) 



Hardware ^ Equipment/ iiiciuc^iag compuJters, disk dfiveSr 

cassette .players', cables, monitors. 

'Materials • , Books, folders J "envelopes, wof4«sh^ets -and --S^ 
• similar items*^ "User Support Materials" -^'re 

items which support the 'activity ot^ person^ 
using a computer progtam. ^ * 

s ^ / « ' 

Nficro . ^ microcomputer. 

■« 

Microcomputer A'computer system, including peripheral . 
^_ ' ' , hardware: such as dislCdr^ve and monitor,, . 

baised on a micrbprocesspr (or "chip") and 
^ ^\ - J having a typewriterrlik^'^keyboard.^ - 

Operating A^progr^on or set of programs', which;controls 

System - and coordinates the operations* of the 

^ componehts of a computer jsf stem. 

Package: One'^or more computer. programs with related 

materials and the storage^m^dium. ^ A package 
repr^^ents a mi<^l:ocomputer *app4^ication. . 



^ ^ Program A computer program^ written in BASICv^ascalf 

machine" code or other computer programiriiTig , 
\ • . language. \ ' 
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(=COrrtiniied) 



C. DEFINITION OF TERMS- 
(Cdnti^nuedJ , ^ 



ROM 



-> 



RAM 



Software 



Storage 



* Read Only Memory: a. microelectronic storage, 
Hjedium whipji: contains a^iirog^^m permanently stored 
there by thq manuf actiifer ^to petform a Specific 

".function.;^ The computer user may onl/ use the^ * • 
^(te program f andtoannpt store a progranC on -ROM or 

• alter a' program /stored, thCre. • * * 

"Random Access Memb'ry: >7fiicroelectronic storage 
medium which ordinarily constitutes the* main 

/working memory , of ^a^ipicrocpmputer* and which is*- . r 
, ^e-usable." That iSy the user of the computer may 
store programs or data i^n RAM^ .and in ^o doing . 
erases information previ-ously stored there., ^ 

Computer programs ^ including application 
programs, operating jsyst^ms, and languages. 

Tape cassette, ffexible disk, ROM cartridge, or r 
for storing/micrqqomguter\programs. 
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Before a package d^an be evaluated, you will 'need sdme ' 
factual Inforraatign about it. for^xample, does it have all 
the cofl^nents necessary to make it an instructionally ' 
usefujl- package? Indeed, determining , the existence of needed 
compQ^erits is itself 'an evaluation. 

The "Courseware Description*; form identifies* the information 
necessary f or evaluatioh -and use of a package. In some 
.cases| a list of components is^incluiped. In a coinplete ^ 
paickage,, all the information should be re^adlly available in*' 
^the program and support materials. ' 

^ Completing the^form, should, fte a strai^bbforward ±ask. If,; 
-hoj^ever, sfi^ie info/mation is not provided in the package, ^ 
you- m^y be aDle to. infer some of it by a trial use of the 
package. • • . . ' . * • 

/ I ^ \ i r * . ^ ' 

Some pf th.e sections on the "Courseware Description" form 
are discussed bfelow. . . ^ . * ^ 



Versidn A v^r^ipn numcfer, ad^te,.or hardware 

j* ' identifier. . . ' ^ . 

Producer , The orig»inal source (developer, publisher, 

' author; individual) Who produced the package. 

Gr.ade/i ^ The.gr^ade^evel, grade, range, -or other 

Ability Level ability level indicator for which the package 

I is ^intended;'' * • 
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Subject Area A general area, such a^ Mathematics, Reading 
or^fHi^tory* ' . ^ ' 

Specific A subset of Subject Area, such as * . 

Topic multiplication, phonics, or World War I* 

, Requited . Identify , the types -of hardware required as a 

Hardware minimum for package lisel Some examples are: 

a. Computer (brand, version) 

b. RAM amolint ' . ' , 
^ c. Mass Storage Devices (disk drives, 

cassettes*) v 
d. Output Devices (a:>lor^or black-and-white . 
monitor or TV, printer) ^ 
, e. Other Peripherals' (joystick, gfaddles, 
voice synthesizer) ' 
^\ f. Special Electix)nics (circuit car<3s, 

iaterfaces) ' ' ' • 
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II • COURSEWARE DESCRIPTION 
(Continued) 



Required Software Identify the language; operating ^system, > 

and utility software required to install 
and use the applicabioji software in thei 

package. Include any drivers; * 

^ . subroutines ot special software^ 

"requirements nof in the* package or. 
' • standard in the hardware specified./. . 



/ 



Instructional 
Objectiives 

Instructional 
Prerequisites 



Lifet the purposes^ .goals or Qbjectives 
that ttie package fs intended to achiisve. 

Describe the experiences, skillsX^ * 
concepts, understandings, maturity\^nd 
ability levels which the intended usei:* 
should possess * for suc9essful use of .the* 
package. . v . ^ , « . 
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- , - ' . r- r \ ' ?.(C6ntl^inued) « 



r \ Describe Package Content'^ and Structure 

\Ab6tract of '^his information should be available in the 
Content material provided by the producer of the ; 

package* However i- you may. wish'- to add 
• ^ comments if the descriptioh is incomplete or 
if^aceurateL, Some suggested components of the 
description 'aire discussed below. 



Genferal ^ . '-Describe the general, content domain of. 
Content the package • - ^ . ^ - , 



User^ -^^scribe the dimension of user control 

Role. *\ ^ over rater sequ^jice, araount, type and content 

of problem examples if applicable. 

> *• ^ • 'J. , , • " 

e.g. ^The pser determines "how much time will; 
oe allowed, for solving e^ch problem. 

; "The user defines the nqmber pf^ 
• ^ hospitals, tfife aroount>of medicine 
available r and the quantity of. J 
pesticides used for mosquito control.* 




Micro SIFT 

EVALUAIOR'S^ 

GUIDE 



,11. COURSfiWARE DESCRIPTION 
< (Continued) * 



Describe- Package Content and StrucJ^re 
(Continued) , ' 

Instriictional Describe the instructional strategy used 
Strategy ' in' the courseware. ' ' • , 



Instructional 
IntegVat4.on 



Program 
Strtrc^re, 



e.g. '•This is a drill alnd practice program 
' designed to increase student proficiency 
and speed in solving quadVadic 
equations. 

' ' * ' i . 

"This gourseware simulates a malaria 
control situation by determining* cost 

^ effectiveness and impact of the 
variables selected by the user." ' 

.Describe the degree i to which the packag^\ 
is or can be an integral 'part of 
in?trtiption. For example, it may be intend'ed 
as a random or casual supplemeiftt, chosen by 
teacher or student ,'^orte of sev^ra^l optipna^ 
On the^ other hand, it may be • the only way' afr 
important topic caA be^.addressed with a 
.student activity. * , ' • 

Describe the;,f low or sequence of 
activity in th^ pr^'gram. *' . 

e.g. "The program contains three* major 

. sections, a |)1:esentation 9f an example, 
\' a section of controlled user activity, 
and* a section of uncontrolled user ' 
exi)loration." • ' 
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The Courseware Evaluation form is designed to be used after the 
information in the Descriptjon form is available. The rating of 
r the 22 items 'in the centerfold of the form is to be the starting 

point of the evaluation. The sections of the form are described 
in the sequence they aze to be completed. 

Judgments should be based on thorough investigation of the 
program and support materials in the package. St is intended « 
that where th^ term ••package" is used, all components are to be 
taken into account in making the evaluation. 



guide '. Ill, COURSEWARJS EVALUATION 

- ^ ^ • ' . (Continued) ^ 



Rating 



Items 1^ through 21 are to be rated .on the scale " 
provided. It is anticipated th$t in' some cases a 
given item may be meaningless for the' package 
being evaluated, and so NA (Not Applicable) should 
be circled. 



«r ^^y-^ 



NOTE: The descriptions of yeach of the items on 
the following pages are iiit^ded -to be suggestions 
fof consideration in arriving at a ^judgement on 
the item. They are not necessarily checklists/ » 

not in ordeif of importance, and are not ^ 
exhaustive of possible considerations. The ' 
alphabetic identifiers are merely for reference 
purposes^ ^ - ' . _ • 



Importance . AIL items on the list .are considered'to be 

"^ important . Hdwever, for a given pacKSige/ certain 
items may tak^ ori greater or lesser importance- 
than most. items in the list. H or L may be 
-circled'^to indic'ate a ^igljer or Ipwer. value for an 
\ * " item in relation to ail» the other items. Since ' 

• , this is an .itidicator of relative value/ not all 
can be identified as H Or L. Use of this column 
may help ypu to apply this set of, critetia to 
packages which dif fer .f rom^'each other in mode of 
instruction. * . ^ . ' '* 

^ \ 

Comments Brief comments concerning the grating of any item* 

may be written in the space ;to the right of the 
' item. ' • 
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^ ' ^ (Continued) 



Student Use It is not expected that student use of' a package 
be observed in- this evaluation process. However, 
if it is convenient, the evaluation may be more . 
valuable. You. as a professional are making a 
judgment based on your teaching experience in the' 
grade level and subjects intended for the / 
.P^ck^ge. If students take part, check the'box at 

-\ (the top^of the center page. ' 
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III* COURSEWARE EVALUAO^'ON 
(Continued) * 



1. The content is accurate > 

i 

\ 'Possible problems^il^ 'ContAit' accujfady Includ * 

outdated information orflastffVctional ^approach 

factual errors ^ ."^ t ' * 

invalid model, usgd in a^Tsipi^ation 

d. ' ^oversimplifie-d model or ex'amp^s ' 
\ , . ' , . ^ . ^ . 7^' 

el improper ase o> stati^tjics - 

^ m^GCurate graphs or (;disp^ay&' . * 



a* 
c. 



7 



4» • : 



'I * 
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A* 
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(Continued) T 

u « — ^J: ^ 

2. The content has educa-tional value ^ . ' ^ * 

.Any decision on this item .will, be highly 
subjective* Some considerations leading to a 
positive judgment might include: - 



ft 



^. The- con£ent and objectives are addressed -in 
common school curricula. ' • 

b. • The knoyledg^ and* sIdLils' involve© have 
uti'lity^in sdSte aspect of life. 

c; An instructional situation^an be envisioned 
in which the package would be useful" ' ^ 



a 



d. ' Use'^of the packagS enables you to leafn 

something about the nature or needs of the 
^ , -student using it. 



e.' The content of the .p^cka^ is central to the 
. * subject field, . * * 



.MicroSIFT ^ ' - . /. . 

EVALUATOR'S 

GUIDE— ^ II I. 'COURSEWARE EVALUATICM^ 

» ^^^pntinued) p 



3 The coQtent is free of 'race^ ethnic and sex stereotypes. . ' 
\ ~ ^ / ". 

'\ ar^ Certain racial, ethni<? or .sex groups jna^ be 

overrepresented at the expense of limiting 
others in their contextual settings. 

.J ^ b. Some racial, ethnic of sex groups may be - 

^ portrayed in 'term:! that are indicative of 
false generalizations about the ^ . 
py characteristics of that group. * 



V. 



MicroSIFT ' » * ' 
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(Continued/ - 



41 The purpose of the package is we'll defined, ^ 

Purposes, goals and objectives may be in the 
program*or in us er^, support m^te.cials,. The 
ideritific^tign of instructional-objectives is 
. ^ , important, to the transferability and use of an 

instructional package. 



a. Objectives should be explicit,, rather than 
. inferred. 

b. Objective statements should be clea'r; i.^., • 
unambiguous ahd without multiple meanings, > 
succinct, 'free, of jargon. 

c. Objectives should be stated in terms of ' 
e)tpected*student behaviors^ 

Th'^ package should include bptri^general "and 
si)iecif ic statements of purpose. That is, ^th^^ . 
overall pirrposeVof the package ought to be"^"" 
conqis^ly stated, *with specific objectives ►stated^ 
for sffecif ip comj^neijt^.. ^ \ " 



t 



MiferoSIFT * • , ' ' '\ ^ • 

^^•XDE III. .COURSEWARE' BVAL^ION • |%.. ^ 

. . • ' (Continued;! . . ' ^ 

~ ' ~' ' [ — ] 

5- The package achieves its defined purposjBw ^ ^ \ ' . 

0, Coursewar^e can be evaluated in much the same way ' 
^'that other inSjtructidn is evaluated — the starting 
• place '•beirfg the instructional objective^.^ B^sed 

on these Objectives, the student using the 
, instructional package -should' Teatn what 'the 

material sets out to teach, rather than merely , . 
being ^engaged un the process. 

^ r - The most .effeq,five wa^ to. sub^$tantiate ''this a^ect 
of instructional quality is through, a sample^run 
of. the pro5-raii(j/ pi^fefably with a learnex ^rom the 
targ^Bt audience, flbweirer, if such a learner is" 
^ not available, the evaluator should make a 
judgement as to- how well t{?e package would — 
actually accomplish its objectives when used by%a ^. 
student of the ' appropriate" /naturity , and ^ability, * ' 
based on the evaluaCor's experienqe w;th- students^ i.^ 
^ of that type. • ^ * * .\ ' . ^ 
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Pack^ge\ Tit|Le 
|^\iLer8ion 
Producer^ 



Subject Atea ' 

Specif icj^pic 

S^ars Subject: Hea'dingS' 
I'ERIC Descriptors 



micTi 




COURSEWARE 
DESCRIPtlON 



Cost 



Grade/Ability Level 



■5 



j^ Mediiim pf Tran^sfer 



_ Tape Cassette ^ --^ , 
ROM. Cartridge 



5" Flex. Disk 
8" Flex. Disk 



i Required Hardware: 



p Required Software : 



Type of Package ; 



Single Program 



Integrated program 
s^eries* \compon^t 



^ Instructional Purpose :^ (Pldase check all applicable) 



Remediation 



Standard Instruction 



Enrichment 



.Ingtructiotfal Techniques: (Please check all applicable descriptions) 

Drill and Practice ' Game 

^ * Tutorial ' • 



Informatiotwri Ret, 



Simulation 
Problem Sblv; 



Learning M§mt. 
Utility 
Other 



Documentation Available; (Circle all that ard available in the <:omnHter 
progt^qpf (P) lor in the supplementary materials !(S))^^ 



P Sr: Suggested grade/ability leveKsh 

P £ ' Instructional Objectives • 

P S P^rerequisite skills or activities 

^ §_ Sample pr^Ogram output^ 

P S^ , Ptog ram x)pe rating instructiojjs 

P S Pre~tfest . ' ' ' 

* Z S ; Post-test' • ■ ft . 



Teacher's information 

P I S ' Resource/reference 
i • information 

P!^ Student's instructions 

P |S Student work8he*et8 

P Relationship to standard 

V ; textbooks 

1 

P S ,^Follow-up activities' 

^P S Other 



r 



Is Ujsting aijid alteration of the comput 



er prograiri' allowed? 
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^Package Title 
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COURSEWARE 
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J- 



Vers? 1 on 
Date 



|25V Describe the pptential use o£ thfe package in classroom settings* 

(NOTE: Complete the centerfold section before completing this item.) 



i]' 




RATING 



3" 
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60 etc 
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§ 



CO 
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SA A D SD 
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SA 



D 



SD 
SD 



SA A D SD 



SA A D °SD 



SA A D ^ SD 



SA A D SD 



SA A D SD 



SA A D SD 



SA A D SD 
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SA A 



sd" 
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SA "^A 



D 

"d* 



SD 

"sb" 
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NA 



NA 
NA 



NA 

"na 



NA 



NA 



NA 

'na~ 



IMPOR- 
TANCE 
(OJ'T) 



u 

0) u 

w) 5 ' 

32 _J 




Northwest 
Regional 
Educational 
Laboratory 



Copyright 1981 NWREL 



Ri^TING: 
IMPORTANCE : 
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L 



H 



L 



li 
ii 
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— -mt- 



H t 



I! 



J. 



□ 



1. The content As accurate. 



Circle tho lottc»-r abbreviation which best reflects yout 
judgment (use the space following each item for comments) . 

Circle the letter which reflects your judgment of the 
relative importance of the item in this evaluation." 

Check this box if this evaluation is based partly on your 
observ ati on of studen^ use of this package . 



Evatuator' s Cuide Page Reference ^ 



2. The content has edutttional value. 



2^ The content is fr^e of race, ethnic, sex, and other stereotypes. 



A. The. purpose of the package is well-defined. 



5. The package achieves its. defined purpose,^ 



6. Presentatiofi of content is clear and logical. , 

7. The level of dj^fficulty is appropriate for th? target audience. 



8.^ Graphics/colorAsound are used for appropriate instructional I'ea 



sons. 



9. Use of the .package is motivational. 



10. 



The package effectively /stimiilates student creativity 



If L 



JNA 
NA 



NA 



NA 



NA 



NA 

'na 



li 
H 

'if 



L 

r 



H L 



H L 



II L 



li 

Y 



L 



11. 
12. 



Feedback on student responses is effectively employed. 



Thi 



learner controls tbe rate and sequence of presentation and review. 



13. 



Initruction is integrated with previous^ student experience."^ 



14. Learning iA, generalizable to^^n appropriate range of situations. 



15. 



The user support materials are comprehensive. 



16. The user support materials are effective. 



17. 



Information displays are effective.. 



18. 



Intended users tan easily and independently operate th^ program. 



19. Teachers can easily employ the package^ 



.20. The program appropriately uses relevant computer capabilities. 



21. The program is reliable in normal use. 



Permission to copy this- 
form for courseware ^ 
evaldation purposes is 
hereby granted. 



. 22. 

CHECK 

ONLY 

ONE 
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1 would, use or recpmmend )ise of this^package with -little or. no change. 

(Note suggestions for effective on the front page..) o ^ ■ 

I would use cir recommend use 6f this package only if certain changes were made. 

(Note changes under major weaknesses on the next page.) < 

I would not use qr recommend this package. (Note reasons under major weaknesses.) 
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2T. Describe tho ninjor sr\iMij',t lis oT Llie packap.o: 




Copyright 1981, Northwest Regional Educational Laboratory , 




Instructional Objectives : « Stated 



□ 



Inferred 



□ 



Instructional Prf?requisites : Stated 



□ 



Inferred 



□ 



■ i 
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Describe Package Content and Structure: 



't 



ERIC 
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Describer's Name Date 



' MicroSIFT ^ ^ ' . ^ ' *; 

GUIDE * . ' i IIl/**COURSEWARE EVALUATION- " ^ % % 



' (Cfontihued) * • - * 



— ^ ■• ; ■ 

6- « presentation of content is clear and locfical , 

- focus of this item is on how the terms, 'facts, 

concepts and principles^of the si^bject mattec are 
. presented father than on ^he content itself. 



ERJC 



a. 'The information i§ well organized. 

b. The structure of the presentation is, evident 
to a user. '"J , 

c. Definitions and explanations are available 
when necessary. ^ 

d. TBfere is a smooth trar^ition between concepts 
and cognitive clusters. * « 

' 1 ' ' T ' 

e. The progression of p'resentation' is logical 
4nd *well identified. . . ^ 

f. E^mp^les/ counter-examples 'and illustrations 
! are' u^ed where possible. . - ^ 

V > ' . ^ ' 

g. The exWples ire relevant to the point of 
instruction. [ ^ ' 



MicroSIFT * / • ' - 

EVALUATuK^S . - ^ j, 

GPIOE , ^ Hi, COURSEWARE feVALUATK^ 

(Cpi\tinued) . ^ 



7» The level of difficulty istfeppropriate for the'target / 
audience ; ' ^ ^ - 

• a. The means of respopse {i.e., multipfe choice, 
' • ' manipulating graphics, single keystroki,' , 

etc,) is appropriate to b?he tajrget audience. 

b. The readability of support materials ahd 
program text is consistent with the. expected 

^ability level of the aydience. Vocabulary, 
^ phrasing and s|hteAce. length are ^specific 
considerationis®^het-e, ^ , \ 

c. Examples and graphic illustrations' a're 

* • . suitable for the mat^brity of the student. 
' • ' ' . ' 

d# The time required for typical student use 
<■ does not exceed the attention span qf the 
. target audience.^ " ! ♦ 



.e. ^ Size of steps in logical processes are suited' 
, * ta the ability leVel of the student, ^' 

f. ^ There are multiple -levels pf vnstruction, 
with diagnostic and reinforcement 'routines, 
for individual differences Tn Che target' * 
. audience. *\ ! ^* . # • 

e.g. The prpgram*";atftomatically branches to 
9 • ^ remediation subroutines if user ' , 
responses require. ' . - r , 



36^. 



J\ MicroSIFT , 
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GUipE,, , 111% cduRSb^ARE 'EVALUATION 



fOontinued) 



The level of difficulty is appropriate for the target - 
. aAJdience. a 



(Continued) 



There are multiple levels of instruction, 
with (Uagnostic and reinforcement routines, 
for individual differences in. the target * 
audience, 

(Continued) / " . * 



The program^ automatically progresses to 
more. difficult problems to continually < 
provide* a challenge to the user that has 
mastered the easier^problems^ 



Thei program automatically provides . 
easier problems to the useif who is 
having trouble. \^ • 



. • MicroSIET . ^ 

\ EVALUATOR'S ^ ' ^ 

• ^ GUIDE III. .COURSEViSfc EVALUATION 

/ » ' (Continued) 



Graphics/sound/corpr are used for appropriate instructional 
reasons , ' • . 

a. Graphics, sound and color enhance rather than 
detract: from the instructional* process. 

' . ' ' ' -J , 

b. Use of souqd does not disturb others in a 
t ^ ^ ' classroom ehviropftaent. ' >^ 

^ 'c. ,Graphics^ sound and color focus att?ention on 
importaric content apeas. 

_ ♦ ^ d. Good message design principles ar^* ysed in 

order to jplace emphasis where important 
concepts arfe. 

^ . ej Visual' and auditory 'effects stimulate student 

V interest. 



.J ^ 



X- 
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llf. COURSEWARE EVALUATION \^ 
'<t:ontinued) 



9. Use 



i 



of the package is motivational 
a. 



b. 



e. 
f. 

g- 



students are effectively addressed in»a 
personal style 1 ^' 

Narr^tive^in the program use humor and a 
conversational manner. ^ - 

The overall tertor of ^interact ion is warm, 
friendly, helpful. 



The paokage. provides for ^ va^iet^ of stu.dent 
. responses and response modes. 

A variety of responses to stu^ent^inputs are 
used. ^ ^ 

Reinforcement is positive arid dignified. ' C 

A student is left with a desire to use the 
package again, or to pursue the topic in - ' 
other ways. " ' ^ 

A student is left with a positive attitude * 
about 'i^he experience.* 



1. 



Using- the package is* a pleasant experience. 



MicroSIPT ' • ^ . ^ ^ 

EVALUATOR'S ' • . . ' T ' ;* 

GUIDE . * , Ilf; . COURSEWARE EVALUATION • ^ 

^ (Continued) / * ' 

~ — : : L " 

iO* ' The package effectively challenges student creativity , 

' ^ • *■ ' ' • ' « ' ' 'J 

^- The learner is involved in an active, rath.er 
. . ' . than passive, manner, in the 'instr taction* ' 

' / ^-9* The student has control over "as ijiany 

. input variables as the -program permits. 

' ' ' / ' ' * " ^The computer i& used in a "hands-on" wai 

• * \ * ratfie^: than ;merely in a presentation' \ 

mode. * " ' • \ 

" . The program design allbws the s/udent as-' 
, mahy ^decisions as possible. / ^ ' 

Tile t)ackag^l provide 6pf»xtunities to answer ' ^ 
-q^nrended questions that have^no ^right" or 
••w1:ong''*^ans\tfej:s, arid^gives the stjident . 
v4 ^ evaluative criteria to judge* his/her own 

^ * responses. ' ' • . ^ 

^ c. , The program, is desig^ied.to anticipate a wide ^ 
range of possible jfe'sponse's.. ^ * ^ 

. d» The student is provided \vjith new ways* of ^ — 
'Ji M , looking at* the world. ' 

^. * /The package demonstrates 4 crfeatdve "means of 
' ,^ using the icnowledge being. acquired by the's , 
* user^. . ' ' . . . ^ ' ' 

J ' - ' ' , ^ , 

f.* The package suggests areas .of further , ^ 
i^^^if^ e;c"jiloration or other activity • - ^ ^ 

^ g. The student , is cha^^lenged to chahge an 
4 underlying modeller design an ^iternative ' - 

, , model* J,' . . ♦ • ; 



, MicroSl'FT 
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GUIDE 
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.(Continued) 



11. Feedpack' on student responses is effectively employed . 

a. The feedback is relevant to tt^e students.' 
responses and: there fol-e "predi'ble".^ 

b. The feedback is noft-thAeatenirig, yet, 
corrective when necessai 



> r 



The f.eedback is timely^ Ke.', givem with 
\ appropriable- frequency aiid\ given inunediately 
after a response* 

The feedt^ck remediates (giyes cues^ hints 
and- explanations) • 

yThere is'^ quantitative feedbac^ when valQable. 

^g. The program* indicates theVhumber and- 
percentage of problems correct out of 
the number of'prdblems att^$Jnpted* 




f . 



g^ 



T ERJC 




• The\feedt?acfc. tells^ "why" l:he^ response was ' • 
incokrect (e.g., "You should have\spelled the 
name correctly." or "Use no punqt^ation.")^. 

iThe* Duqgemeiit of student responses properly 
assesses the concept being taught, not merel^i^ 
its foriA.* . . - ' 



e.g. Is word^otrder roor« i!i4)ortant .thaA\the . 



c6nte\it*ot the ^jfrspohse? 



The programXadapts to the learnet by 
adjudting the difficulty level of content) 



29: 
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<n (Continued) ' ' 



12« The leanier controls the rate and sequence of presentation 
and rev iew. 

" » ■ ' ■ ■ « * 

a. Student has control over the time allowed for 
' ' . - ■ solving problems, allowing for quickening or 

slowing theu pace as. the user deetns necessary. 

( ^ b. Student has control over the' rate of 

presentation of .display .materia?, so that 
- he/she can read and absorb the information at 
' \ his/her own rate. 

*c. The program does not lock -the student into a 
J ' ^ linear instructional sequence. 

d. The program allows the student to begin at a 
point appropriate to his/l?er past 
. V acl^ievemertts. * ^ 

• e» The program: has a provision fo^' review of 
' instruct i'onUss initiated by the user. 

<; The progr^defirles "functions" fqr learner* 

* ' * options such as "|HELP", "HINT", "DICTIONARY." 
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(Continued) 



13» Instruction is integrated with previous student experiences . 

Instruction is designed to take into account 
the background experiences typical of the 
target audience. 

Inductive, reasoning is employed. Known 
•situations are used tb exptain new "'Situations. 

Conunojily> experienced examples are used. 

■f . ■ • . , ' ■ 

^.g. Some students. may better understand 
liquid metric measurements withijn the 
context of filling the car with g&soline 
rather. than filling a graduated cylinder 
w^ith water. ' 

Instriictioa. moves from the concrete to the 
abstract, simple to conplex,^ familiar to 
unfamiligir. - 




ERIC 
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(Continued) 



Learning is ggfieralizable to an appropriate range of 



situations. 



Thfe^fearning is applicable to a student's 
future experiences'. 

e.g. The'instruction prepares. the* user for 
the next unit in the package. 



b. The studenV is presented with opportunities 
that require generalizatiorr of the rules 
acquired at the computer and opportunities to 
apply those rules to ''real life" situations 

'away from the computer. . * 

c. The processes and information learned aire 
useful in domains and situations other than 
the subject area of. the packag&. 

d. ' The content is orgatTized in such a -way as to 

facilitatie recaliyand Japplicatibn away from, 
the^ compurter and/4o*g/L<4 of - the immediate 
content, dbmainr 

e.g. Is the. metric- system taught within the 
context of the decimal system,, or as 
isolated measurements/ (meter, gram, ^ 
' litet, etc.) . . * ' 



; ERIC \ • : >^ 
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(Continued) 



15* User support materials are comprehensive ♦ - ^ \ - 

yi In thi^ item, you are assessing the completeness*^ 
of the package, in terms of i,ts support £ot the 
tfeachers and students in the intended pattern of 
use, and reasonable optional uses. Many different 
typesL^of^mfqrmation may be included in the 
Jfirited material accompanying the program. The 
components of good user support materials 
identified here can be packaged in mainy ways; 
Separate^ identification does not imply a need for 
separate booklets, although that may be . 
desirable. (See also item 16 •) 

a. Student materials ; sufficient materials for 
a variety of student activities should be 
provided • — ^ 

• * * , 

pre-instruction activities relating to 
the package 

a guide to %se of the package 

follow-up -activities to reinforce the 
•instruction 

AC 

• — wprksheetQ 

Teacher's Information' 



a desfcription of the instructional 
activities to take place 



MicroSIFT 

EVALUATOR'S . ^ 
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(Continued) * > 



*15# User'support roaterials are coroprehrfnsive . 
(Continued) . * 

* . • b. Teacher's Information 
(Continued) 

Suggestions for classroom logistics in ^ 
4^ vapety of hardware situations (single 

or multiple Machines, hardware not in/ 
classloom, etc.) 

. ~ -a rationale for computer use | 

• . • — prerequisiteTskills necessary for best 

utilization .. ~ * : 



teacher directed pre and post 
instructional activities, 

. Resource Information 

bibliography of rebources and references 
related to the content .domain 

' • ■ ■ . .-x — • 

seuiple run of. the program ^ , 

jfcssibilities for program modifications 



a description of the model u^ed in 
simulations ' 
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III. COURSEWARE- EVALUATION 
(Continued) 



15. User ^support materials are comprehensive , 
(Continued )^ 

d. Technical Documentation 



~ , detailed explanation of how ttig^program • 
and package operates,. 

program code listings * - ' « 

explanation of u6er definable options to 
adapt the program for different . 
.applications. 

explanation of the software/hardware 
interface or any other extraordinary 
•fe^tkres of the. program 

flowchart or other diagrams or general 
logic if .individual programs and package 

interpretation of error messages 
Containers ' 

folders, binders, pocketsf for storing 
printed mat^ials, disks, cassettes. or 
other components 

— ^ boxes or othpr container for organizing 
'and storing^ the entfire package 



MicroSIFT . , ' * 

EVALUATOR'S ' . _ 

GUIDfe- ... V III. COURSEWARE EVALUATION 

(Continued) • 

lJ ^ 

16* The user support materials are effective d 

5*^ "^^^ appearance of* the. materials i^ attractive. 

b. The quality of the papei^ or binding is, 

. s 'appropriat;e to its. intended use and expected 
. ' \ ' • . life. i . 

c. The printed text- is clear, readable and 
attractive.* < . 

• d. Pi^ctures, diagrcrtns and^ graphs are appropriate 
and readable. 

e. The text, captions, labels, etc, are 

thoroughly edited, and free of errors in 
grammar, spelling and punctuation. 



f . The packaging of the materials is suitable 
for the inten^e^ use. 

e.g. -student worksfieet masters indented for. 
reproduction are "loose leaf* or easily 
^ reproduced. - . 

■ - / '* — 

teacher support -Tnaterials can- be 
separated' from stuae'nt materials. 

g. The program-storage media are e^s^ly 
accessible, yet protected frojd p^ncJom injury 
expedted in ^nailing, dropping, etc. ^ 

h. ^ The entire package is storable as a \iirit in . 

standard storage facilities {of fiqe, shelves, ' 
• • cabinets, etc.) , ' 



Micr< 
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(Continued) 



16'. The user support materials are . effective . ' , 
(Continued) Z ^ 

,> ^ > 

\. Materials are easily used in table spaie 
* typically available aear, a microcomputer 
station. 
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1*7» Information displays are effective *' ^ ' \ 1 

Good message design principles are/incorpota.ted 
into tn? visiwl arrangement of disblay material* ^ 

- ^* Graphic di^piayd ace not too/conjplex or fulj^ 

i of ^too much 'infori^tj^on. - ^ 



'There is -ladeguarte spacing on the screen 
* * or printed materials fjot clarity.- 

Static and dynamic, gi^^phics are used 



when a|>pli cable* - 



'0 



0' > 

f ^ 



4' 4nd prinCea^^'^splays tiake 

^ Mfe^ivfe^ Qs^^ of fpjfea space. " 

^^ol^ Y^ Text^ftarr*feive pp the^ m^i|lt;or or printe^^ i^ 
^ clear and^ ^asy to rea^./ ' * ' 

<?... Ka^rative not i^^aious. ^ . ( ' 

d.' Text Infortn^feipn is Qoi^te^lengthy* br^ , 



' ' " V ' id®^ ia> given ajS^uate. time to read and.. 
'4 ^ f informatfion siven^n the ^displays. 



' ''^Tj^,?|/fr€e fro^ punctuation 



> ^ g. pfiar^tjit^-^et^^^ for' 



.c 
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n*^ Information displays are effective ^ 
(Continued) - . » 



h« , Graphics axe, not too repetitive or too slow 
in presentation. . . 

i% Input options are inctependent of color, ot at 
least avoid dommon cotbt blindness problems* 

J. There is nc^ too much text for the display. 

The, te?ct position is»consistent And/or 
predictable (i»e., the student does not have 
to hunt for the information) . 

k. Graphics are appropriately mixed with text 
material to give variety to the £)resentdtlon. 

i. Transitions from dfsplay to display on a ^ 
video scteen aire ^mootjh and unobtrusive. 

m. Scrolling is used appropriately. Only 
pertinent information is retained on the 
screen, i • - ' . 

n. Adequate teacher/student options for^use or 
nonuse pf Sound are provide(}> 

o. * Graphics are not distracting to the user . 
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III. COURSEWARE EVALU^ION 
(Continued) 



IS. 



Intended users can easily and independently operate the 
program^ \ ] ' ~ 

a. Th^ program has enough^lnternal. documentation 
to permit ease of use even without external 
.paperv documentation. 



b. 



Format0\and protocols for user-computec 
/ comraunicjation are consistently applied and 
logical.,^ 

Directions ^e aggompanied by useful examples 
where^appropisjiate. % . >^ ] 



d. 



Help pages and functions are provided and 
accessible at liicely points of\ieed. 

e. The program does not^^^allpw the user^to get 
lost iri the program\?ith/no apparent wa 
out. The student alwtfy^^has some options for 
getting th^ program runiiing again, or 
returning to. a beginTlin|K point • ^ 

* ' ' • * \\ ' . 

f . The program doesn't stop or appear to be - 
" doing nothing without clues\ . ^ 

qr Traps are used copiously to c^cb- potential 
^errors of any kind, and to avoid moving / 
^control from the application i6o.\he operating 
system software • 

ii^ Instructions and error ihessages are\:lear and 
unaunbiguous . They give tHe user cleai 
directions as to what he/she must do tc 
effectively use the program. 
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Intended users can ^^MBr^^j^Q igdependently operate the 
^ prograrR^ > / 
(Cojitiriued) 
' ^ ' S • y * V 

i. The program re;5^onds tp inputs as the 
directions indicate. ' 

j* The liser can -easily exit the program, return 
to menQs,^or move to another section with ^ 
program-described conventions* * 

° / k. ' The program accurately evaluates jstudent 

input; i.^., it does not misinterpret student 
responses and thereby identify a response as 
incorrect when it is in fact correct. 

• 1.. Computer operation* does not inter^fete with 
concentratipn on the activity. 

m. The program .can be used with a minimum of 
computer competencies. « <> ^ - 



n.. 



the-'u^er isn*t/ uneasy using the sbfware dUe ^ 
to its complexijty of oj^<^t^ion. • * * 



^ o^ The user rs informed qf' .which fuQction keys 

he/she will use in the coi^rse of ^tKe program 
^ • a^d' their ' pihrpose. - sQ. ^ 

V 

There is th^ necessary cueing for function 
key usage. . * • ^ ' 

'^..^ Those fllmfct ion 'keys referred to in fche^ 
' . program are available on the tiardware^ 

^* The tise of function Ijeys does not necessitate 
re-input of> user responses previously Input 
' • ^ * * into the computet. 

^ > " * ^ ^ ' 41 ' f/™- 
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19. Teaoherg can easily employ ^the package . 

Not only s{)ptild .the program be easily used by. the ^ 
^ . / studen^^^if^ut it should be equally em^Jbyable by ' • 
thfe 'teachers. Many of the same considerations as * 
in item 19 pan be applied here» but also: 



a^^ The program can b^^ used by a person Having a 
^ minimum of computer 'competencies. . 

^ ' ' , ' * ' / 

b. ^ ^ The program requires a raihimal amount of 

equipment manipulation bV the .^te^ch€»r. 

c. ' Software modification? or uni^ual ' 

manipulations^f disks are notN^quired to 
effj6c}:i^e'Ly use the program. 

d. The-package is easily adaptable to a varie%y 
of classroom 'learnittg environments, including 
placement of hardware inside^ or outside the 
<ilassroom. • ^ 

e. Error handling and identification are 

. sufficiently detailed so ^jxe t^ache^ can ' ^ 
' easily, help a stu^en^. * ) 

f'. Student^/ require a mi^nimum amount of teabher 
supervision while usiiig the progr2^m. / * • 
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2lO« Tne program appropriately uses relevairt computer V 

capabilities .^ - ^ * 

^•^ ■" * * 

' ' • • '\ * 

*Thfi success .of the computer as a means for - 

, instriSction is due to those capabilities inherent^ 

4 the t-echnology. Computer software should take • 

full adva^ntage^ of . the unique aspects of the 

^ * • . computer rather than jnerely doinop the same 

.,1 activities in; ^ new way. - ^ 

• ^» T^ie application is well suited, to computer . 
^ . , use and not one that can Ise ^handled more 
'-^ . .> appropriately by/other .means. . , 

' \ s b. Course management or. computer .collection and 
' i organi;sation/pf data-, op instruction i^* • 

• * . 'available. ^ " , * ' . 

— ^ \/ ^' V ^ ^ > _ . ^ V . ' ' ' i ' / ' 

. ,;v^ - , -e.g^ The^:|riformatidn;al5p ^ 

; perfojfitiance .is" stored for retri<5val .at a ^ 

^; ' -< ' J' * later, time. * ^ • - 

• • * e»b..^^The ^computer is used in a dynamic/ . ^ . 
- interactive way.'-* * * : . 

fe.g^'» The /domputei; makeE decisions! based on 
» /Student performance accordihg tdi the 
* ^ ?;teacfiing^stre|tegies' inherent; to. i^^ 

d;; * ^he computer n^ke^ effective .use of , ot:her 
V . ^.c/^ /jP?f^^P^iera%^^e^^ printeiys^. Xi^ht ^ / 

;4 V' ' ,P®^^Si ;''padt^e contr^^^ fetpO ; 

. / * ! ^^fo^^af'ltje^nate in^^ 
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20. The^ program appropriately uges relevant computer * , 
* capabilities ; ^ '~f ~. ^ 

(Continued)" ' 

, * , ®* -^^^ computer is used to, simulate activities . 

\ , • , ttiat are too difficult, dangerous, or 

^ expensive to demonstrate in reality. 

f. .The computer ife used so that students .are 

' . • actively involve^ in a ^hands-on" xnannei;: ^ . 

, - • • ^ rather *'thati only pas^jfvely observing. . 

g. Th,e computer responds to natural student * 
input 3uch as ."YES" or "NO"; or their first ( 
letter rather than "1 = YES, 2 = NO.^ 



III. COURSEWARE EVALUATION 
(Continued) 



21 ♦ The pcoqram is reliable ir/ normal use . 



The program will consistently run under all 
' normal conditions, N(vspecial precautions 
such as clearing memory are required" for 
effective program execution. 

The program will consistently load into the 
computer without* undue complexity, such* as 
re-loading'. a • 



T^e program is,ftee of programming and^ 
operation.al errors ("bug-free")* 
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22. Evaluator Recommendation . - > 

- Having concluded a review of the ^instruct io^l 

objectives, prerequisite activities, content and 
/ ^ . structure, and fating the various considerations 

of content, instruction and technical -Quality, you 
are then ready to make a recommendation as to the 
ufie of the package. 

You have three options ,f tom which to choose to . . 
represent your .estimation-of the program's 
usafbility. These options cprr^spond to section. 23 
of the Evaluation Form. Elaborations can be made 
in sections 24 through 26, which ask for ways the 
program can be used effectively and^ identification 
of major strengths and weaknesses in the program. 



a . Recommend I would -use or recommend use of this 
Progrart / prograro/pactcage with little or no "~ 

' , change. (Note suggestions for effective 
use, next. page.) p , 

b. Recommend I would use or gregommend use of this 
With Changes program/package only if certain changes 

we'te made.. (Note changes under major 

weaknesses, next page.) 

*' . ^ * 

c. Do Not I would' not use this program/packages 
Recommend (Note reasons under major weaknesses. ) 
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23. Describe the major strengths of" the package > 

24, Describe the major weaknesses of <^the package. 

In completing these item's the evaluated should .refer to 
Content, Instructional, and Technical Quality ratings 1 - 
through 22. These ratings, along with any notes or 
conunents that the evaluator made, should serve as* the basis 
for specifying the program's str-engths and weaknesses. 
Fbllowing are some example comments. 



Content Strengths 



"The laboratory simulation was 
accurate and could be. used in 
place of the» real laboratory 
situation. *• 



Content Weaknesses 



Instructional 
Strengths^ ' . 



"The instructional'' objectives 
are. stated and are ^ very 
specific as to terminals 
performance behayioi 

"Some-equations were graphed * 
icorrectly on the monitor. " 

"The recommended audience a^e 
is too. young for program 
content." 

^Tbeuser has control over many 
of the variables affecting 
his/her instruction." ^ 

^> ' 

"The simulation adequately ■ 
considers all of the variables 
related to the problem." 



-Rir 



47; 



59 



MicroSrPT • 
BVALUATOR'S 

GUIDE I IIV COURSEWARE 'EVAEURTION- 

" (Continued) 



23. * Describe the major strengths of the, pafckage, 
I ..TcontTnued] 7 Z ' 

^ . - *i 

24. Descriipe the major weaknesses of the package* 

(Continued) . ^ - 

, Instructional 6 ."(Phe program, does not allow students 

Weaknesses to generate their own answers — only 

mak^ a choice o^ four options 

•The student only^^at^ph^^s the 
program. , Ther;e is^ no active 
, ^ participation in the courseware. 

. Tectinicaj: o "o^he directions were^ very clearly 

Strengths stated/ giving examples when 

necessary.** 

* ' \ ^ 

^ o "The documentation is thorough and 
. comprehensive. 

..." 

Technical ^ ^ o "The computer was uged ^s nothing 

.Weaknesse^^ V V more than an electronid^Wextbool^. 

( , None bf th^ uhtgue interactive 

t characteristics were ijSsed. " » 
. " . - ' ^ • • » ^ ^ 

o "The. documentation included only a ; r 
listing of the jprograllir* ' Np 
^ • objectives or prerequisites were 

mentioned." 
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25. Describe the potential alse of the package in clas3ro|^ " « 
settings ^ * . . ' 

• 

The list of items 1-22 in the centerfold of the ^evaluation 

form shduld be cpmpleted t>efore this item. ^ ^• 

rfecoraraendation for use with or without changes should be 
accompanied with a description^ in this spaqe of the 
possible use you envision for the package in specific 
class roo|n settings.* The description mig*ht inclyfJe: 

, * a. The different audience characteristics to 

whom the program is applicable. " 

e.g^The package should **be used with 
individuals, and small groDps only 

Program oould also' be useawith . 
audiences older than the rAdmipended 
target audience. - , - 

i , h.^ The different' teaching modes the program may 

enhance. 

'* e.g. The package .can be used to introcluce the 

topi^.s ' ' . t 

The courseware can be used to reinforce * 
related instruction and to test the 
' / * adequacy of that instruction. . ' . 

c. Any "unique**- applications the courseware may 
, A have. / . w ' ^ 

. , ^ e.g. ThA package could be used in a "contest" 

. / . ' ^ forrt to see whic^h team can devise the 

✓ best strategy for solving the problem. ^ 

ERIC . . 



